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ABSTRACT 



A formative evaluation was conducted to determine the types 
and amount of dissemination and services the Pacific Regional Educational 
Laboratory (REL) provided to its constituency in the first 9 months of 1999. 
The evaluation also studied how recipients of REL products and services 
perceived the usefulness and quality of the products and services provided, 
and it identified strengths and areas for improvement for program refinement. 
A total of' 129 questionnaires (39%) of an original 300 were returned, and 
most were usable. In the first 9 months of 1999, the REL disseminated over 
8,000 copies of products, print and other media, to its constituents, and 
over 75% of the recipients found the products useful. They expressed 
satisfaction with the overall quality of the materials, indicated that the 
products had a positive impact on their work, and encouraged the REL to 
continue to improve the accessibility of its products to educators. In the 
same period, the REL provided about 60,000 person hours of service in the 
various formats. Response to these services were favorable. Over 50% of 
respondents received the services available to them, and most did so more 
than once. The REL conducted 21 workshops and presentations, and an 
overwhelming majority of participants ranked the attributes of the workshops 
and presentations very highly. About 20% of survey respondents were not 
familiar with REL products, a finding that suggests that ways should be found 
to promote REL products and services and improve their accessibility. Three 
appendixes contain the client survey, the workshop participant questionnaire, 
and a list of products distributed. (Contains 19 tables.) (SLD) 
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EXECUTIVE SUMMARY 



The Pacific Regional Educational Laboratory (REL), a major program within Pacific 
Resources for Education and Learning (PREL), serves Pacific children and educators in 
10 U.S. -affiliated entities in the Pacific region. The REL primarily conducts 
development and applied research that directly contribute to broad-based school 
improvement to help all students meet high performance and content standards. At the 
same time, the REL works closely with partners such as educators, researchers, parents, 
and policymakers in planning, implementing, and guiding the development of new 
information, knowledge, and programs to improve educational practices at school and in 
other educational contexts. 

This formative evaluation has three purposes. First, the study summarizes the types and 
amount of dissemination and services REL provided to its constituency in the first nine 
months of 1999. Second, the study examines how recipients of REL products and 
services perceive the usefulness and quality of products and services provided. Third, the 
study identifies strengths and areas for improvement for program refinement. 

A total of 129 (39%) questionnaires, out of an original 330. were returned. The majority 
of the questionnaires are complete and valid except two with no responses arid a few with 
illegible handwriting in the open question sections. 

In the first nine months of 1999, the REL disseminated over eight thousands copies of 
products, print and other media, to its constituents. Most respondents were familiar with 
REL products and used the materials in their work, and over 75 percent of the users 
found the materials useful. They expressed their satisfaction with the overall quality of 
the products, and indicated that the products have a positive impact on their work. In 
addition, respondents encouraged the REL to continue the good work and to improve the 
accessibility of its products to educators, especially at school levels. 

In the same period, the REL provided about 60,000 person hours of services in the 
various formats. Responses to REL services were favorable. A large portion of the 
respondents were familiar with the REL services, with workshops and training as the 
most recognized form of services. Over 50 percent of the respondents received the 
services available to them, and most did so more than once. They found REL services to 
be useful and have noticed the impact on their work and their students’ performance. For 
future REL service provision, respondents primarily recommended that REL expanded 
the content coverage of its services. 

The REL conducted 21 workshops and presentations, mostly in Honolulu and in Saipan, 
which served as the host site of the 16 Ih Annual Pacific Educational Conference. An 
overwhelming majority of the participants ranked the various attributes of workshops and 
presentations they attended very highly. The responses indicated that the content of the 
workshops and presentations was appropriate, content coverage adequate, and ideas 
presented useful. A common theme of the comments is a request to allocate more time to 
these activities since there is so much to learn. 
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An average of 20 percent of the respondents was not familiar with REL products, 
including the most widely distributed newsletters and brochures. Ways should be 
explored to promote REL products and improve their accessibility to a broad base of 
educators. One of these ways is to seek assistance from service centers in the region in 
both promoting REL products and updating the REL mailing database. 

REL services to the region, including products and services, were highly praised, 
individually and collectively. To make its service to the region more effective, the REL 
should, based on the needs in the region and its strengths/specialty areas, develop series 
of workshops around a theme, and coordinate the production and dissemination of 
products and technical assistance. 

Introducing new knowledge and promising practices to the region via workshops and 
presentations is the beginning of a process, and follow-up activities are necessary to 
secure positive changes. The REL should consider developing means to measure the 
changes occurring over time in terms of implementation and outcomes. 
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INTRODUCTION 



Background 

The Regional Educational Laboratory (REL), a major program within Pacific Resources 
for Education and Learning (PREL), serves Pacific children and educators in 10 U.S.- 
affiliated entities in the Pacific region. They are: American Samoa, the Commonwealth 
of the Northern Mariana Islands (CNMI), the Federated States of Micronesia ( FSM) 
(Chuuk, Kosrae, Pohnpei, and Yap), Guam, Hawai'i, the Republic of the Marshall 
Islands, and the Republic of Palau. These ten states are referred at entities because of 
their very different forms of political affiliation with the United States. 

These ten entities cover a vast area of almost five million square miles in the Pacific, with 
a total population of 1.3 million. The level of economic development varies from entitv 
to entity. 

Schools in the Pacific are challenged not only by political and economic diversity, but 
also by cultural and linguistic differences. At least nine distinct Pacific cultures are 
prominent in the region. Although the language of instruction is English; for most 
students outside of Hawai'i, it is not their native language. 

In Hawai'i, there are 179,876 students in 240 public schools, while the other nine entities 
have about 98,000 students in 343 schools, according to the data collected in the 1993-94 
school year. 

REL works closely with other federally funded programs under the parent organization 
PREL, a nonprofit organization that provides the overall governance and infrastructure 
that support educational programs and service to children, educators, and schools in the 
region. Other major programs are the following; 

• Pacific Mathematics and Science Regional Consortium 

• PRELStar: A Pacific Islands Distance Learning Program 

• Pacific Comprehensive Regional Assistance Center 

• NEARStar: Network for English Acquisition and Reading, A Star School Project 

• Pacific Vocational Education Improvement Program 

• Freely-Associated States Educational Grant Program 

• Pacific Parental Information Resource Center 

• Pacific Center for the Arts and Humanities in Education 

PREL has service centers in American Samoa, Honolulu, Saipan, Guam, the Marshall 
Islands, Palau, Pohnpei, Chuuk, and Yap. Other centers will be established in the 
remaining entities to build a region-wide network of service centers. 
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REL Objectives 



REL’s program of work is guided by the objectives of the REL program derived from 
Title IX of the U.S. Elementary and Secondary Education Act (ESEA). Primarily, the 
REL conducts development and applied research that directly contributes to broad-based 
school improvement to help all students meet high performance and content standards. 

At the same time, the REL works closely with partners such as educators, researchers, 
parents, and policymakers in planning, implementing, and guiding the development of 
new information, knowledge, and programs to improve educational practices at school 
and in other educational contexts. 

REL Programs 

Nineteen ninety-nine is Year Four of the program’s second five-year contract, and the 
REL continues to focus on the following functions to achieve the objective of supporting 
education in the Pacific: 

• Strengthen governance, management, and planning; 

• Conduct applied research and development; 

• Participate in the formation of a nationwide system of education information and 
assistance; 

• Collaborate on development and dissemination services; 

• Assist the U.S. Office of Educational Research and Improvement as requested; and 

• Conduct development, applied research, and dissemination in the specialty area of 
Language and Cultural Diversity. 

To perform these functions, the REL established these task forces: Governance, Planning 
and Evaluation; Applied Research; Production and Dissemination; Pacific Data Center; 
Scaling Up; Technology; and Specialty Area: Cultural and Language. 

The broad range of services performed by the REL to its constituency falls into two 
major service categories: production and dissemination of information, and training and 
technical assistance. 

The REL keeps abreast of the educational innovations and research materials that respond 
to needs expressed by Pacific educators. These, along with educational publications and 
products resulting from original research work by REL staff, are disseminated throughout 
the Pacific region and shared nationwide with other regional educational laboratories, 
educational agencies, and other interested organizations. The products that the REL 
produces and disseminates to its constituents encompasses a broad spectrum of media 
including newsletters, briefing papers, research syntheses and reports, audio/video tapes, 
and websites. 

A significant part of REL services to the region is as convener and trainer in conferences, 
workshops and seminars, technical assistance, and partnership collaboration. The REL 
facilitates exchanges among diverse groups across entities with a focus on specific role 
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groups, special interest topics, and continuing initiatives. In addition, the REL provides 
various forms of technical assistance in response to the needs of local communities. The 
workshops and presentations cover topics such as school reform, at-risk students, 
evaluation, school management, and literacy. 

Purpose of the Survey 

Formative evaluation is conducted during the operation of a program to provide program 
directors or managers with evaluative information useful for program improvement. 

The purpose of this formative evaluation is three-fold: 

• First, the study summarizes the types and amount of dissemination and services that 
the REL provided to its constituency in the first nine months of its fourth contract 
year. 

• Second, the study examines how recipients of REL products and services perceive the 
usefulness and quality of products and services provided. 

• Third, based on the review of survey results, the study summarizes the strengths and 
areas for improvement to help program directors and managers to refine future 
program planning and implementation. 

The information derived from this formative evaluation could be used to improve 
development of applied research, materials production, and services delivery as they are 
being planned, implemented, and maintained to support regional and local educational 
reform efforts. 
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METHODS 



Instrument 

The 1999 Annual REL Products and Services Survey instrument was modified based on 
the one used in 1998 Annual Survey, in accordance with the objectives of the REL and 
the intent of this year’s formative evaluation study. (See Appendix A.) The instrument is 
comprised of two parts, a cover letter and a survey form. The cover letter from the 
Director of Planning and Evaluation gives survey respondents a brief introduction to the 
survey purpose and the goals of the REL, and requests their cooperation in completing 
and returning the forms within a specified time frame. 

The survey form includes three sections. The first section contains two demographic data 
fields: entity, where the respondents are from; and role, their professional capacity. 

The second section was designed to check whether respondents were familiar with each 
of REL products in the following categories: newsletters, policy briefings, research 
papers, conference information, audio/video tapes, and websites/pages. Examples were 
provided for each category to remind respondents in case they were unsure of which 
product the category referred to. If they were familiar with them, the next section asks 
respondents to indicate how often they used the product over the past nine months by 
choosing one of the four levels: four or more times, two to three times, once, or none. 
Further, if they did use the product at least once, respondents were asked to indicate how 
useful each product was for them in their professional work, again with four levels: very 
useful, moderately useful, a little useful, and not useful. 

The third section lists six statements about the products as a whole, regarding the overall 
quality, their contribution to respondents’ professional knowledge, skills, work, decision- 
making and planning, and finally, the impact on student performance. For these items, 
respondents have five choices: strongly agree, agree, neutral, disagree, and strongly 
disagree. 

The two open-ended questions are for respondents to elaborate on the aspects they like 
about the REL products, areas for improvement, and areas not covered by the items in the 
two tables. 

An identical format was used to study REL services including workshops, training, and 
technical assistance: respondents’ familiarity with, frequency of using, and usefulness of 
services in the first table; ranking of quality, contribution, and impact in the second table; 
followed by open-ended questions for strengths and improvement. 




PACIFIC RESOURCES FOR EDUCATION AND LEARNING 



Page 6 



Specific evaluation questions related to REL products and services addressed by this 
formative study include: 

® What products and services are REL constituents aware of? 

® How do REL constituents use the products and services? 

* How are REL products and services useful? 

• What judgments of quality do our constituents have regarding the products and 
services? 

® What are the best things about REL products and services? 

® How could the REL improve its products and services? 

Sampling 

PREL has maintained comprehensive client databases tracking individuals, who over the 
years received products and services from the REL and other programs. Also collected is 
information on the intensity of services received in terms of hours. In the services 
database there are over 1,000 non-duplicated names, with accumulated service units 
ranging from 30 minutes of phone conversation to more than 10 hours of intensive 
training. 

According to past experience, a person who only received a copy of newsletter or had a 
one-hour information session with REL staff was unable to give meaningful feedback on 
REL products and services. To address this issue, the survey focused limited resources 
on a sample that is smaller but more knowledgeable about REL products and services. 
After examining the distribution of service units of those in the database, it was 
determined that those with more than a total of two hours of services should be included 
in this study. This amount of service allowed surveyors to have a pool large enough to 
take care of the “no returns,” while ensuring that respondents had a reasonable amount of 
exposure to the products and/or services to be able to make informed choices and offer 
meaningful comments and suggestions to improve future REL products and services. By 
using this criteria, a sample of 240 participants was extracted from the database, which 
served as the mailing list for this study. 

Data Collection 

A PREL envelope containing a cover letter, a survey form, and a self-addressed and 
stamped return envelope was mailed to each of the individuals on the mailing list. They 
were requested to complete and return the survey form in about two weeks time after 
receiving the package. In entities where PREL has established a service center, center 
staffs help was enlisted to distribute and collect survey forms. Barriers of distributing 
and returning survey forms were encountered in the region in the previous studies, largely 
due to the remote location of schools, related transportation and communication 
difficulties, and different postal service systems in the Federated States of Micronesia, the 
Republic of the Marshall Islands, and the Republic of Palau. To alleviate the negative 
impact on the return rate, in addition to direct mailing, service center staff were requested 
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to distribute an additional 90 survey forms to other recipients of REL products and 
services who met the above mentioned conditions. 

Data were stored electronically and a quality control process was in place to ensure that 
data were coded and entered accurately, i.e., random visual spot check and computer 
check for outliers. 

Analysis 

Data analysis was performed using SPSS windows statistical package for IBM personal 
computers. Most of the statistics presented in this report are descriptive in nature, i.e.. 
frequency distributions for most of the data fields. In some cases, means and standard 
deviations are reported. These statistics are presented in tables in their respective 
sections and efforts are made to make them comprehensible and meaningful. 

Respondent Profile 

A total number of 129 (39%) questionnaires, out of a total of 330, were returned by the 
end of November, the deadline for data entry. Most of the questionnaires were complete 
and valid, except two with no responses and a few with illegible handwriting in the open 
question sections. As a result, the number of responses in each field might not always be 
identical. 

The following is the profile of the survey respondents. Table 1 indicates that of a total of 
127 respondents, 28 percent were from Hawai'i, 23 percent from the Federated States of 
Micronesia (FSM), 17 percent from Palau, 13 percent from Guam, 10 percent from 
CNMI, 5 percent from American Samoa, and 4 percent from the Republic of the Marshall 
Islands (RMI). Among the four states of FSM, most of the respondents, 86 percent, were 
from Yap. 



Table 1: Number of Survey Respondents by Entity 



Entity 


Respondents 


Percentage j 


American Samoa 


6 


5 ! 


CNMI 


13 


10 i 


Guam 


16 


i3 : 


FSM 


29 


23 ! 


Chuuk 


1 


i ! 

i 


Kosrae 


2 


2 ! 


Pohnpei 


ill! 

* i 


Yap 


25 ; 


: 19 

I 


Hawai‘i 


35 


! 28 

1 


Marshall Island 


5 


4 


Palau 


23 


17 


Total 


127 


100 i 
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Table 2 reports the breakdown of the respondents by professional role. It shows that 30 
percent of the respondents were administrators; 36 percent teachers; and 21 percent 
specialists. Students had a small presence in the respondent pool. 2 percent, and so did 
the other group, which included two school aides and one curriculum staff. 



Table 2. Number of Survey Respondents by Role 



Role 


Number 


j Percentage 


Teacher 


46 


36 j 


Specialist 


27 


21 1 


Administrator 


48 


39 


Student 


3 j 


2 


Other 


3 ! 


2 | 


Total 


127 | 


100 | 
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STUDY ONE: EVALUATION OF REL PRODUCTS 



Products Disseminated 

During the first nine months of 1 999. from January I to September 3 I . the REL 
distributed a total number of 84,982 copies of materials to its constituents. (See Appendix 
C for listing.) The total student enrollment of the 1 0 entities is 276.99 1 . according to the 
data collected in 1993-94 school year; on the average, the RJEL disseminated 
approximately one copy to every four students. A large portion of these materials, 

43,080 copies, went to the Hawafi educational system, which had the largest student 
enrollment among all entities, 179,876. The copy to student ratio for the State of Hawaii 
was about one copy per every four students, close to the region-wide average. Although 
other entities received fewer copies of materials, the copy to student ratios were higher 
than that of Hawai‘i because of their relatively small student enrollments. Yap and Palau 
had the highest ratio with more than one copy per student, CNMI and Kosrae, about one 
copy for every two students, and American Samoa, Chuuk, and Guam had similar ratios 
to Hawai‘i with about one copy for every four students. (See Table 3.) 



Table 3. Number of Items Disseminated by Entities 



Entity 


Items 

Disseminated 


Student 

Enrollment 


Items per Student 


A. Samoa 


3,690 


14,375 


0.26 


Chuuk 


4,089 


17,650 


0.23 


CNMI 


4,769 


7,710 


0.62 


Guam 


8,338 


30,417 


0.27 


Hawai'i 


43,080 


179,876 


0.24 


Kosrae 


1,567 


2,546 


0.62 


RMI 


3,500 


11,096 


0.32 


Palau 


3,454 


3,317 


1.04 


Pohnpei 


3,362 


8,798 


0.38 


Yap 


2,041 


1,206 


1.69 


Subtotal 


77,890 


276,991 


0.28 


Other 


7,092 




Total 


84,982 



Note: Student enrollments are for public schools (1993-94) only. 



About one third (32%) of the materials were sent to classroom teachers, including 
elementary, intermediate/middle, high, and other types of schools. Nearly one fourth 
(27%) of them went to curriculum specialists. The next two largest groups of recipients 
were District or Central Office administrators (13%) and school principals (9%). Faculty 
of higher education institutions, universities and community colleges accounted for 8 
percent of the dissemination materials, while students, parents, and community members 
had a share of 5 percent (see Table 4). 
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Table 4. Number of Items Disseminated by Role Group 



Role Group 


Items 


Percentage 


DOE/Central 


10,685 


13 


Specialist 


22,572 


27 


Principal/Vice Principal 


7,440 


9 


Student/Parent 


4,529 


5 


Teacher 


27,210 


32 


IHE Faculty 


7,084 


8 


Other 


5,462 


6 


Total 


84,982 


100 i 



Survey Findings 

Familiarity with REL products 

The large quantity of products that the REL disseminated were grouped into six major 
categories: 

• Newsletters 

• Briefing papers 

• Research syntheses and reports 

• Audio/video tapes 

• Information about conferences and meetings 

• REL websites/pages 

The first section of the survey form asks respondents to indicate whether they are familiar 
with each product category. Table 5 shows that most of the respondents, 78 percent, 
were familiar with the REL newsletters, Pacific Education Updates, which was the most 
widely distributed in the region through mass mailing. Seventy-two percent of the 
respondents reported familiarity with information about conferences and meetings such 
as the Pacific Educational Conference, also widely publicized. In the next two product 
groups, 47 percent and 42 percent of the respondents indicated familiarity with research 
syntheses and briefing papers, which were content/area specific. Unlike the previous four 
categories which use print media, the last two groups require additional hardware to 
utilize the rich information contained in the products. About a third of the respondents 
(36% and 31% respectively) were familiar with the REL website and audio/video 
products. 
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Table 5. Familiarity with REL Products 



Products 


Familiar 1 Not Familiar ! 


Newsletters 


i 78 j 22 I 


Information about conferences and 
meetings 


| 72 j 28 | 

f f 


Briefing papers 


40 ; 60 


Research syntheses and reports j 


47 | 53 : 


Audio/video tapes 


3 1 | 69 


REL website 


! 

o 



N = 127 



Frequency of Use 

Table 6 shows how frequently respondents used materials disseminated by the REL. if 
they were familiar with them. The sample size for each product varied as a result. 
Among those who were familiar with the product, 90 percent used newsletters and 
information about conferences and meetings at least once, 85 percent or more for 
audio/video tapes, research syntheses and reports, and briefing papers, and 67 percent for 
websites. Again, the use of websites/pages was limited by the availability of computer 
and internet capability, especially in entities other than Hawai‘i. 

Information about conferences and meetings was the most frequently product used: 28 
percent of respondents reported that they used it more than four times. Newsletters were 
the second most frequently used product with 24 percent for four or more times. The 
REL distributed newsletters and conference brochures more than any other products, and 
people used them to find out information about conferences and meetings, and the 
happenings in the Pacific education community. 



Table 6. Frequency of Use of REL Products 



Product 


Frequency of use 


4 or more 
times 


2-3 

times 


Once 


Never ! 

i 

i 


Newsletters 


24 


51 


15 


10 


Briefing papers 


17 


43 


25 1 


! is ; 


Research syntheses and reports 


10 


41 


35 


14 

( 


Audio/video tapes 


20 


37 


40 


13 i 

1 


Information about conferences and 
meetings 


28 


37 


25 


10 J 

i 


REL website 


13 


1 31 


23 


13 ! 

J 1 



1 6 
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Usefulness of Products 

The percentages reported in Table 7 were calculated based on the number of respondents 
who were familiar with and used the products. Again the sample sizes for each category 
varied from one to another. An overwhelming majority of at least 85 percent of 
respondents found the products very useful or moderately useful. Over 50 percent of the 
users indicated that briefing papers, website, information about conferences and 
meetings, and audio/video tapes were very useful, 46 percent for research syntheses and 
reports, and 27 percent for newsletters. 



Table 7. Usefulness of REL Products 







Usefulness 


] 


Product 


Very 

useful 


Moderately 

useful 


A little 
useful 


j Not 
i useful 1 


Newsletters 


27 


49 


18 




Briefing papers 


58 


32 


\ 1 

! 6 

i 


4 


Research syntheses and reports 


46 


41 

1 


11 

i 


2 

i 


Audio/video tapes 


55 


39 


6 


0 


Information about conferences 
and meetings 


55 


31 


8 

i 


6 1 


REL website 


56 


28 


16 _J 


0 : 



Ratings of Products 

Table 8 summarizes the responses to the six items in the survey form from those who 
were familiar with the products as a whole. Fifty-six percent of respondents agreed that 
the quality of REL products was high, with an additional 30 percent strongly agreeing. 
Over 90 percent reported that the products had contributed to their professional 
knowledge. A slightly smaller portion indicated that REL products had contributed to the 
improvement of their professional skills (73%) and work (75%). About 70 percent 
agreed that they used the products to inform decision-making and planning, while slightly 
over half (51%) of respondents observed that products had positively impacted student 
performance in many ways. 



17 
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Table 8. Ratings of Attributes ofREL Products 



Evaluation Item 


Strongly 

Agree 


Agree 


Neutral i Disagree i ^ tron £^ ■ 
l j Disagree i 


The quality ofREL products is 
high. 


30 


56 

i 

j 


1 13 | 0 I 

1 i ' 

! i i 


REL products have contributed to 
my professional knowledge. 


29 

! 


1 62 

i 

1 1 


1 4 1 3 : 2 

| i 


REL products have contributed to 
my professional skills. 


24 ! 


49 

1 


20 i 5 ! 2 ! 

I ! . 


REL products have contributed to 
my professional work. 


25 


50 


17 


; 6 : 2 i 

j i i 


I have used REL products to 
inform decision-making and 
planning. 


21 


48 


25 


5 i 1 i 

1 

i ! 

1 

! ; 


REL products have positively 
impacted student performance. 


16 


35 


41 


7 i 1 i 
1 



Comments 

"What do you like_ about_ the REL products " 
New Information 



The majority of the respondents expressed emphatically that they valued the body of 
knowledge and information delivered in REL products. For many educators in the region 
REL products are one of the main venues to update their professional knowledge and 
follow the latest development in the field. Without the products it would be either 
impossible or at great cost to obtain the information essential to improving their decision 
making or classroom practices for the educators. The following are some typical 
responses: 

• The latest information on teaching strategies. 

• Very informative. Keep us updated on current events. Keep me abreast of what’s out 
there. 

• [P]REL products are very informative. 

• Update on latest information, share research, reports, and papers. 

• Informative and resourceful. 

• Keep me informed of the trends in education. 

• Good source of information, and an ideal way to communicate to each entity. 

• I like reading the Pacific Education Updates because it is the only way of learning 
what is happening at PREL in regard to workshop, training, programs and teaching 
tips from schools and teachers of other places, mostly from Hawai'i top teachers. 
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The products that are available are of a wide variety. P, K-12 and generally high 
quality. 

Easy to follow, to the point, research based. 

® Like the professional work. 



Relevant and Practical 



One of the strengths ofREL products evident in survey responses is that the information 

delivered addressed their educational needs and the new ideas were applicable in 

classroom teaching and decision making. Some responses indicated that the promising 

practices would result in improved teaching practices and student performance. 

• So far the REL products have positively impacted student performance. 

• REL products have contributed a lot to my professional knowledge, and work in any 
subject areas. 

• Its diversity and applications 

• Several teachers are subscribed to the REL products for enhancements. 

• It helps educators to improve their performance, and, in return, students’ learning 
improved. 

• It helps students succeed. Or I should say, “want to succeed”! 

• Research syntheses and reports because they relate to our Pacific clientele and mesh 
importantly to our culture . 

• Concise and practical. Can (be) put to immediate use. 

• Address the needs of professional developers to. . .impact school improvement and 
thus student achievement. 



Focus on the region 

This point, closely related to the above one, was emphasized repeatedly in the survey. 
Respondents particularly appreciated the fact that REL focused on the regional needs in 
producing and distributing culturally appropriate products that educators and students in 
the region could identify with. 

• The appropriateness and relevancy of the products — focus on the region. 

• Research that applies to our entity. 

• Appropriate to local areas. 

• Targets Pacific Island needs and issues. 

• You are doing a good job. It is sometimes that our entity doesn’t take advantage of 
your service. 

• Research-based articles to support Pacific regional issues. 

• They address local issues and topics. Information is relevant to local situations. They 
also compare/contrast with information from other national sites. 
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Suggestions for future improvement 
Keep up with the good work 



Many respondents indicated they had nothing to offer for this part because they were 
satisfied with the current REL product dissemination, while others wrote to encourage 
REL to keep up with the good work they had been doing. Some suggested that the REL 
re-evaluate the needs in the region, and then produce and distribute quality products on a 
variety of topics to address the urgent needs. 

® Continue on as you have done the previous years. 

® Evaluate to make sure that the products fit the needs of our students. 

® Include variety of innovative approaches and strategies that would appeal to 
classroom teachers. 

• More research on Micronesia. 

• More product diversity affecting learning process of Pacific (students). 

® Follow up in the entities on how they use those materials. 

• Provided a series of products connected to each other. 

• Do more research on certain problems that we face today and give us more 
explanation about it. 



Easy Accessibility 

Quite a few respondents reported they knew nothing about REL products distributed in 
the region until they received this survey, and would like to learn more about them and 
receive some REL products. The titles and examples provided in the survey form had 
already attracted their strong interest. 

• What are they? How can I obtain one? 

• Not disseminated to program office. 

• Make them more accessible to teachers in the entities. Maybe distribute them to 
specific departments or spending time meeting with departments on their needs and 
inform them about your products. 

• It would help if schools can be exposed to the other products that REL has to offer. 1 
only got one. 



Strong Promotion 

To make REL products more accessible to educators, especially at the school level, a 
strong promotion effort is necessary to raise public awareness of REL products available 
for the community and to improve the distribution mechanism connecting REL and 
educators in the region. 
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° Keep publicizing the availability of these publications. Do more publicity for the 
website. 

9 Put every teacher on the mailing list of PRJEL. It will ensure that all teachers will 
read, know, and understand the benefits of all these products and services. 

® Make sure to send all the professional information to all participants. 

9 Broaden access and distribution. 

® Have available a listing of resources and where to find them. 



\ 
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STUDY TWO: REL SERVICES 



Services Rendered 



REL provided a range of services in the region in the following categories: 

® Workshops and training; 

® Training of trainers; 

® Technical assistance; 

• Conferences and meetings; and 

• Collaborative partnerships. 

During the first nine months of 1999, a total number of 57,401 person-hours of service 
were provided to REL constituents. The unit of service, person-hours, was calculated by 
multiplying duration of an activity with the number of participants. In terms of student 
enrollment, services were provided at a rate of about one half of an hour per student 
enrolled. Among the ten entities, the State of Hawaii accounted for a significant amount 
of person-hours. However, with an equally larger portion of student enrollment, the 
average hour of service per student was about one tenth of an hour (six minutes), which is 
below the region-wide average of 1 1 minutes. American Samoa, Guam, and Chuuk had 
similar person-hours per student as Hawaii. Compared with these four entities, Kosrae, 
RMI, and Pohnpei had slightly more services per student, ranging from 12 to 20 minutes. 
In contrast, Palau and CNMI both received more than one hour of service per student (see 
Table 9). 



Table 9. Number of Person-Hours by Entity Compared with Student Enrollment 



Entity 


Person-Hours 


Student Enrollment 


Person-Hours per Student 


A. Samoa 


1,487 


14,375 


0.10 


Chuuk 


3,022 


17,650 


0.17 ’ 


CNMI 


10,667 


7,710 


1.38 


Guam 


3,566 


30,417 


0.11 


Hawaii 


18,023 


179,876 


0.10 


Kosrae 


800 


2,546 


0.31 


RMI 


2,415 


11,096 


0.22 


Palau 


3,452 


3,317 


1.04 


Pohnpei 


2,109 


8,798 


0.24 


Yap 


1,190 


1,206 


0.98 


Subtotal 


46,731 


276,991 


0.17 


Other 


10,670 






Total 


57,401 







Note 1. Student enrollments are for public schools (1993-1994) only. 
Note 2. Person-hours = number of persons x hours contacted. 
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Table 10 reports the distribution of service hours by role group. The largest role group 
served by the REL was teacher, which included teachers of elementary school, 
intermediate or middle school, high school, and other types of schools such as K- 1 2. 
This group received 45,621 person-hours of service from REL, 32 percent of the total 
volume. Eleven percent of service hours went to the specialist group which inc ludes 
curriculum specialist and resource teachers. A similar portion (12%) was accounted for 
by administrator of DOE/Central Office and a smaller one, 8 percent, by the group of 
principal and vice principal. Faculty of higher education ( 1 0%) and professional 
association (2%) received a combined 12 percent of the total sendee hours, while the 
student/parent/community group reported a portion of 4 percent. The Other group, 
including regional service providers and federal agencies, received a sizable portion of 
services, 20 percent. 



Table 10. Number of Person-Hours Contacted by Role Group 



Role Group 


Person- Hours 


Percentage 


DOE/Central 


7,303 


13 


Specialist 


6,245 


1 1 


Principal 


4,386 


8 


Teacher 


18,411 


32 


IHE Faculty 


6,169 


10 


Student/Parent/Community 


2,047 


4 


Professional Association 


1,007 




Other 


11,833 


20 


Total 


57,401 


100 



Note: Other includes regional service providers, federal agencies, etc. 



Survey Findings 

Table 1 1 reports how familiar respondents were with the types of sendees that the REL 
provided in the region. The majority of the survey respondents indicated that they were 
familiar with REL’s various types of services. Of the five types listed in the survey, 
workshops and training were the ones that most of the respondents were familiar with, 
with 89 percent reporting affirmative, followed by conferences and meetings at 76 
percent. About two thirds of respondents knew about training of trainers (67%) and 
technical assistance (65%), while 58 percent reported familiarity with collaborative 
partnerships. 
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Table 1 1 . Survey Respondent’s Familiarity with REL Services 



Service 


Familiar 


! Not Familiar 1 


Workshops and training 


89 j 11 ! 


Training of trainers 


67 


i ; 


Technical assistance 


65 


35 

1 ! 


Conferences (i.e., Pacific 
Educational Conference) 


76 


24 

! 

1 


Collaborative partnerships 


58 


! 42 

J ! 



N= 172 



Of those who were familiar with REL services, over 90 percent attended workshops and 
training in the past nine months. Almost an equal percentage (36% and 37%) attended 
once or two to three times, while 19 percent attended training four times or more. A 
smaller percentage, yet still more than half, of the respondents utilized other types of 
services: collaborative partnerships (56%), training of trainers (58%), conference (61%), 
and technical assistance (65%). Most of them utilized the services once or two to three 
times. (See Table 12.) 



Table 12. Frequency of Using REL Services 



Services 


Frequency of use 


4 or more 
times 


2-3 times 


Once j 


Never 


Workshops and training 


19 


36 


37 


8 


Training of trainers 


5 


26 


27 


42 

1 1 


Technical assistance 


6 


33 


26 


35 , 


Conference (i.e., Pacific 
Educational Conference) 


6 


27 


28 


39 i 

1 

t 

1 


Collaborative partnerships 


11 


23 


22 


44 j 



An overwhelming majority of the service participants (80% or more) gave positive 
responses to all five types of REL services on a four-point scale, from very useful to not 
useful. Over 50 percent of respondents reported that the services they received were very 
useful and more than 20 percent reported moderately useful. Among the five types of 
services, workshops and training received the highest ratings, with 76 percent indicating 
very useful and an additional 20 percent reporting moderately useful (see Table 1 3). 
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Table 13. Usefulness of REL Services 



Service 


: Usefulness 


j Very j Moderately 

useful j useful 


j A little ! Not 
i useful ! useful [ 


Workshops and training 


76 


i 20 i 2 1 2 : 

J 1 ; 


Training of trainers 


67 


i 21 


: 6 ; 6 


Technical assistance 


58 


r 26 


| s 1 6 : 


Conference (i.e., Pacific 
Educational Conference) 


58 


32 


5 ! 5 

| i 


Collaborative partnerships 


57 


25 


7 i 11 



When asked about the overall quality of REL services, the respondents gave a very clear 
message: 52 percent agreed and an additional 35 percent strongly agreed with the first 
statement that “The quality of REL services is high.” Responses to other items 
demonstrate a similar pattern. Over 80 percent of them reported that REL services had 
contributed to their professional knowledge, skills, and work. A smaller yet still 
substantial portion of them indicated that REL services had positively impacted their 
decision-making and planning and student performance, which is the ultimate goal of 
REL services. (See Table 14.) 



Table 14. Ranking of Attributes for REL Services 



f 1 

Evaluation Item 

| 


Strongly ' 

! ! 

Agree j 


i 

Agree j 

j 


1 

Neutral j 


Disagree j 

| 


Strongly ; 
Disagree 


The quality of REL services (i.e., j 

technical assistance and training) is i 
high. 


j 5 j 

i 

! 

i 


52 | 

i 

1 

1 


1 1 


2 j 

i 

j 

i 

i 


0 : 

i 


REL services have contributed to 
my professional knowledge. 


42 


46 


10 


2 ! 

j 


0 : 


REL services have contributed to 
my professional skills. 


33 


47 


18 


2 i 

! 

j 


0 | 


REL services have contributed to 
my professional work. 


36 


46 


17 


i i 

l 

i i 


0 j 

i 

i j 


REL services have impacted my 
decision-making and planning. 


25 

I 


48 


24 


! 2 ! 

i 

I 


i ; 

i ■ 

j * 

i i 


REL services have positively 
impacted student performance. 


| 28 


36 

i 


31 


| 5 
! 


i 0 

t 

1 ; 

i J 
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Comments 



“What do you like qbouj the REL sendees? ” 

Quality Services 

Almost all those who responded to this question indicated that what impressed them the 
most is the high quality of REL services. This assessment was also reflected in their 
responses to other attributes of services such as content coverage, cultural sensitivity, 
local relevance, presentation style, and the professional staff, some of which will be 
elaborated in the following sections. 

• In terms of workshops and training, I mostly like how they are set up. Workshops 
have people who really know what they are talking about — experienced teachers. 
Their experience I can relate to. 

• Training and workshops: it gives teachers more ideas to deal with strategies in 
classrooms. 

• Provide the high standard of learning to students and myself. 

• Professional. And good follow-up system. 

• Very good systemic procedure - practical and hands-on. Simple to understand and 
teach. 

• The workshops on various areas: lesson planning, screening, assessment, teaching. . . 

• Services are accessible, appropriate, and well planned. They are useable for teachers, 
trainers who work with schools. 

• REL services helped teachers in updating with new trends in education. 



Professional Staff 

Staff who delivered REL services played a vital role in meeting the needs of their 

constituents. Responses show that educators in the region gave high marks to REL staff, 

for their professional knowledge, extensive experience, understanding of the local culture 

and needs, and their willingness to travel extra miles to serve the community. 

• Friendly, competent trainers. Trainers have left no stone unturned in their efforts to 
assist our school reform. 

• Good presenters with a lot of experience. 

• Consultants are familiar with Pacific Islands. They know how to train Pacific Island 
educators. 

• They find the most competent or qualified trainers to do their training. 

• The willingness of experts in the field have been very helpful in providing training 
workshops for our schools both on-island and off-island whenever they are available. 

• Nice staff, people knowledgeable. 
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Relevance 



One prominent feature of REL services is its relevance to the needs of its constituents. 
REL services provided participants with opportunities to gain knowledge of promising 
practices and new skills they can actually use in their work: policy making, school 
administration, or classroom teaching, and to know somebody they can rely on for further 
consultation and assistance. 

• Reliability, applicability, relevance. 

• Relevance to our needs and purposes. 

• Able to accommodate when assistance is needed. 

• Reach out to meet the unique needs of island type education as well as Western 
schooling. 

• Sensitivity to its constituents. 

• It connects Pacific Island educators. 

• REL services are relevant for Micronesia Islands. 

• They are tailored to meet the needs of local school/programs. 

• Relevance! Almost all of the training is immediately applicable and relevant for 
making teaching and learning in the classroom better. 



Collaborative work 



Another feature that respondents liked about REL services is the collaborative approach 
that REL adheres to in planning and implementing services. By working together with 
educators in the region, REL has been able to offer useful services relevant to the local 
needs. 

® The collaborative work among entities like the PCIC. 

• Collaborative in approach and delivery. 

• Supports system initiatives. 

• Involvement with M/S, R&D Cadre, and PCIC. 

• Collaborative efforts. 

• Promote collaboration and joint planning and implementation. 



Impact on work 

The goal of REL services is to positively impact service recipients in their work, and 
through their work, on children’s learning. Most of the responses reported encouraging 
outcomes of REL services, ranging from gaining new ideas to improved teaching skills, 
and to networking with other educators. 

• They improve my skills and knowledge. 

• Training and workshops, it gives teachers more ideas to deal with strategies in 
classrooms. 
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o Provides greater contributions in enhancing students’ learning abilities. Also REL 
services assist teachers in their everyday lesson planning. 

® REL services helped teachers in updating with new trends in education. 

® I like the Pacific Conference the most. Because J have a good chance to meet other 
educators from other places and share knowledge among educators. 

® Resources of inspirations 



Suggestions for ijrgiwyemen]. 

As the previous section shows, respondents were generally satisfied with REL services 
for the high quality of services, professional staff, local focus, and collaborative 
approach. Many respondents who were familiar with REL services encouraged REL to 
further improve its services to the region in various areas. In addition to providing more 
services, they suggested REL cover more content areas, have follow-ups after initial 
services, and build local capacity. 



More workshops 

• More services to the neighboring islands. 

• Provide workshops and training to all teachers, especially the new hires. 

• Extend the workshops to all schools in our entity and work collaboratively with our 
curriculum specialists. 

• More on-site visits. 

• Make resource people more available for teacher training. 



More content coverage 



• Make an impact study and outcome evaluation of SAT 9 Test results from California 
that is presently being administered in CNMI. What is the substance compared to the 
curriculum? Does it provide accurate evaluation for student profile? Can REL 
develop an evaluation for Pacific Islands? 

• Strengthen health education component and increase training opportunity to all the 
service entities. 

• Offer more training to administrators. 

• REL could improve its services by conducting more seminars and workshops for 
teachers particularly in Reading and Literature. 



Follow-up 

• Follow up on implementations. 

• Provide more follow-ups. Review how implementation of specific teaching 
techniques went. Do not disappear. 
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Q 



Collect impact data and follow-up data, impact on teaching/instruction, impact on 
content, impact on assessment. 

0 Do more research on certain problems that we face today and give us more 
explanation about it. 



Local capacity 

• Highlight local strength! And build local capacity. 

® Make sure higher education faculty participate in standards training so we know what 
is expected for teachers. We will then apply them to teacher training programs. 

• The training should be long enough for trainees to team what they should learn. 

Train more local educators to be trainers, so they can help consultant when they 
conduct training. 



Miscellaneous 



• Keep shaping the technical advice, training and workshop to fit our needs. 1 
appreciate the assistance I have received in Drug Free school and computer training. 

• More services and information geared for Hawai’i educators. It is hard for Hawai'i 
teachers to attend four hours that are so much geared for Pacific islands teachers. 
Although we can benefit from each other’s cultures, etc. Our teachers often need 
different kinds of help and focus. Offer practical workshops and technical assistance 
for Hawai’i schools. 

• Provides lists of services available and schedules. Indicate how each entity can get 
the services and the related costs. It would really help if services were given to 
entities free. 

• Not just lecture type and training but need to incorporate more hands-on activities to 
clarify meaning. 
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STUDY THREE: EVALUATION OF WORKSHOPS AND 

PRESENTATIONS 



Workshops and Presentations Conducted 

Twenty-one training workshops and conference presentations were conducted during the 
first nine months of 1999, including 14 training workshops during the period and seven at 
PEC (four pre-conference institutes and three conference workshops). The following is a 
listing of workshops and presentations: 

® Enchanted Lake Elementary CSRDP Awareness 

• CSRD Kea‘au First Step Awareness 

• Kapalama School Reform 

• Powerful Units 

• Radford High School Workshop 

• Leadership Institute 

• Leader: A Change Agent 

• Tattooing Aloud Instills Learning 

• Comprehensive School Reform, 

® Wai'anae High School Standards Workshop 

• Reading Success Network Institute (3) 

® Strategic Teaching and Reading at Niinakuli 

• Ebeye Literacy Development 

• STRP/CSR 

• English Literacy for Second Language Learners 

• Enhancing Teaching and Learning 

• Program Evaluation 

• Parent Involvement: Read Aloud, Key to Literacy 

• Project Exceptional 

• Comprehensive School Reform 

• Common Sense Parenting 

Findings from Evaluation 

Every training or event convened by REL was evaluated through the use of a 
standardized participants evaluation instrument — the Presentation/Workshop Participant 
Questionnaire. (See Appendix B.) Staff who conducted a workshop or made a 
presentation collected participant evaluations. In most cases, presenters and trainers 
facilitated the workshop evaluation data collection by directly distributing questionnaires 
to the participants at the end of the workshop or training. In some cases, workshop or 
training participants mailed the evaluation questionnaires to REL. 
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This questionnaire includes eight evaluation statements and four open-ended questions. 
For the statements, respondents are asked to rank the workshop or presentation on a five- 
point scale from “Strongly Agree” (5) to “Strongly Disagree” (1 ). 

More than half of the workshops and presentations, 12 out of 21. were conducted in 
Hawai'i. Seven conference workshop presentations were delivered in Saipan (CNMI). 
host of the 16 th Annual Pacific Educational Conference. In addition, CNMI and Kosrae 
each hosted one workshop. (See Table 15.) 



Table 15. Number of Workshops/Presentations by Location 



Location 


Number 


j Percentage 


Hawai‘i 


12 


! 57 


CNMI 


8 


1 38 ] 


Kosrae 


! L 5 1 


Total 

i 


21 


i ioo i 

_j j 



Three hundred and ninety-nine participants attended the 21 workshops. Corresponding to 
the number of workshops held in different entities, 236 participants (59%) were from 
Hawai‘i. The next largest groups of participants were from CNMI and Guam, 33 (8%) 
and 23 (6%) respectively due to the fact that PEC was held at CMNI, and these two 
entities sent large contingents to the conference. The rest of the seven entities each 
accounted for a percentage ranging from 2 percent to 4 percent. When the workshop 
participant numbers compared with the student enrollment of each entities, again. Yap. 
Palau, Kosrae, and CNMI became the ones with the largest participant to student ratios 
(0.004 or larger), while the rations of other six entities were about the same as the overall 
average (0.001). (See Table 16.) 



Table 16. Number of Workshop/Presentation Participants by Entity 



Entity 


Participants 


Percentage 


Student Enrollment 


Participants per 
Student 


A. Samoa 


14 


4 


14,375 


0.001 


Chuuk 


17 


4 


17.650 


0.001 


CNMI 


33 


8 


7,710 

l i 


0.004 


Guam 


23 


6 


30,417 j 


0.001 ! 


Hawai‘i 


236 


59 


179,876 


0.001 


Kosrae 


10 


3 


2,546 


0.004 


RMI 


15 


4 


11,096 


0.001 


Palau 


13 


3 


3,317 


0.004 


Pohnpei 


13 


3 


8,798 


0.001 


Yap 


8 


2 


1,206 


0.007 


Subtotal 


382 


96 


276.910 


0.001 


Other 


17 


4 


j 




Total 


399 


100 


1 
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With 180 participants, 45 percent of the total, teacher , again, was the largest role group 
followed by DOE and Central Office administrator at 1 7 percent, principal and vice 
principal at 16 percent, and specialist at 13 percent. Twelve parents (3%) also attended 
REL workshops and presentations. (See Table 17.) 



Table 17. Number of Workshops/Presentations Participants by Role Group 



Role Group 


Participants 


Percentage 


Teacher 


180 


45 


Principal 


63 


16 


Specialist 


51 


13 


Administrator 


68 


17 


Parent 


12 




Other 


25 


6 


Total 


399 


100 



Note: Twenty-one workshops were evaluated. 



Participants’ overall ratings for the 21 workshops were positive. Of the 399 participants, 
250, over 60 percent, ranked the workshops they attended as “Excellent,” 26 percent as 
“Good,” and 9 percent as “Average.” Twelve participants (3%) assigned low rankings of 
“Poor” and “Very Poor” to the workshops they attended. Combining the first two groups 
of “Excellent” and “Good” ratings, almost 90 percent of the respondents had a favorable 
impression of REL workshops and presentations as a whole. (See Table 1 8.) 



Table 18. Ratings of Overall Workshop/Presentation Quality 



Rating 


Respondents 


Percentage 


Excellent 


250 


63 


Good 


102 


26 


Average 


35 


9 


Poor 


8 


2 


Very Poor 


4 


1 


Total 


399 


100 



Note: Twenty-one workshops/presentations were evaluated. 



Table 19 reports the mean ratings for the workshop attributes based on participants’ 
responses to each attribute on a five-point scale. All mean ratings were larger than 4.00, 
and most of them close to 4.50. The first item: the appropriateness of the workshop and 
presentation content to the region, had the highest mean rating of 4.56. The sixth item, 
the workshop’s impact on the region, was given a lower rating of 4.39, which was still 
well above the mid point of 3, with a possible range from 1 to 5. 
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Table 19. Ratings for Attributes of Workshops/Presentations 



Attribute 


Mean Rating 


Standard 

Deviation 


Appropriate to region 


4.56 


0.43 


Appropriate to work 


4.53 


0.45 


Content Adequacy 


4.46 


0.51 


Usefulness 


4.52 


0.49 


Likeliness to recommend 


4.44 


0.55 


Impact on region 


4.39 


0.77 


Meeting expectations 


4.51 


0.52 


Overall quality 


4.47 


0.51 



Note: Twenty-one workshops/presentations were evaluated. 



Comments 

‘What do you like about the content and/or the style ofjhe presentation of this 
workshop/presentation ? " 

Stimulating ideas 

The first question gave the participants an opportunity to say what they like about the 
content/style of the workshops presented by REL staff and its sub-contractors. A review 
of the responses reveals that new and practical ideas presented at the REL workshops and 
presentations is the number one reason participants liked the presentations. The 
following are some typical comments: 

• It is a subject that I need to address in my classroom but have a hard time finding 
materials. 

• Very stimulating ideas. 

• The content was relevant to my job that needs critical analysis and understanding. 

• Gave us things to ponder — good to revisit ideas presented earlier. 

• Eye opening. 

• Theory can be directly related to classroom implementation. 

• Lots of great ideas, and new mental tools to develop learning in the classroom. 

• This is a “shot” in the arm — a booster that should last until next time. 

• Relevant philosophically as well as practically. 

• The content and style of the presentation are remembered and will be used back 
home. 

• Great synthesis of information. 
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Effective presentation 



Workshop and presentation participants liked the ways presentations were delivered for 
their engaging style, audience-oriented and balanced format (lecturing, group discussion, 
and hands-on activities), and energetic and knowledgeable staff who made workshops 
instructive and enjoyable. Participants always appreciated the effective use of media 
technology, i.e., video clips, slide shows, and good handouts. 

• Effective presentation style, because of his use of examples of media documentation, 
fast pacing, super storytelling ability, sense of humor, and excellent handouts. 

• Boring presenters are deadly -- this team was very motivating. 

• The presenter was dynamic, entertaining and extremely informative 

• Energetic and charismatic. 

• Student work samples were great. 

• Hands on materials and real life discussion. 

• The presentation style is relaxed - just right for Pacific Islander participants. Both 
teams were sensitive to the participants needs which made this whole week every 
enjoyable. 

• The material was presented at a good pace with enough reinforcement. 

• Good organization, able to look at a variety of materials in a time saving manner. 

• A good combination of lecturing, group discussion, and hands on experience 

• Presented material logically and thoroughly. Made a difficult to understand topic 
very comprehensible. 

• Great use of visuals that add to the lecture - PowerPoint, overhead and videos 

• Nice combination of lecture, video, and computer slides. 

• Video clips. 



Suggestions regarding the content and/or the style of this workshop presentation. 

Allocate more time for workshop presentations 

A strong voice from responses is that these workshops and presentations were of high 
quality. At the same time, they indicated that these activities can be improved by 
allocating more time to them so that participants can really learn and understand the 
information and new ideas, enabling them to use the skills in their future work. 

• Well presented but the time was short. 

• Keep content and style, but in a two to three day workshop, (instead of one.) 

• Would like to have more time to review the entire program. 

• Keep style, and content of presentation, but in a two to three day workshop. So much 
information to cram into one day. 

• Plan for presentation better and allow for more time for teachers to look at material 
for the various components. 
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o 



Would have been more effective if she has shown more video clips and/or other 
examples of her project. Let’s have more of it next time. 

Shorter days with more days rather than 8:30 to 4:30. 



Miscellaneous 



In addition to having more time allocated to workshops and presentations, participants 
also want to see improvement in other areas such as the use of media technology and 
handouts. 

• Continue this effort and develop this into a series. 

• Can be more effective to show more video clips and or other examples of the project. 

• Less lecturing and more talk among us during breakouts. 

• Have materials and outlines prepared ahead of time. 

• The transparency might be incorporated in the PowerPoint presentation. 

• Transparencies might be better incorporated in the PowerPoint presentation using a 
scanner so we can better view them. 

• Prepare the reading material in advance. 



What should be the contenj o[_ the next workshop as a (ollow MR !o this workshop'^ 

No specific content areas for the next round of workshops/presentations were identified. 
However participants did express clearly and loudly that they desired more workshops in 
the same content area, but at different levels so more educators in the region can attend 
and then apply the knowledge and skills in their work. 

• Same content, but more time. 

• More advanced skills training. 

• Conducting workshops pertaining to each entity’s needs. 

• Instructional strategies to beef up results. 

• Unit lessons that will help to engage and motivate slow learners in the classroom. 

• Teaching techniques and strategies. 

• Integration between vocational education and core subject areas. 

• Some good/practical alternative evaluations. 

• One to one reading instruction. 
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SUMMARY OF FINDINGS AND RECOMMENDATIONS 

Summary of Findings 



RJEL Products 



In the first nine months of 1 999, the RJEL distributed over eight thousand copies of 
products, printed and other media, to its constituents, mostly in the Pacific region. A 
majority of the respondents was familiar with each of the five categories of REL 
products. Of those, most of them used the materials in their work, and over 75 percent of 
the users found the materials useful. When queried about the overall quality of REL 
products, they overwhelmingly expressed their satisfaction with the products as a whole, 
and indicated that the products had improved their professional knowledge, skills, work, 
and decision-making. More importantly, over half of them reported that the products 
have improved their students’ performance. Respondents encouraged REL to continue 
the good work and to improve the accessibility of its products to educators in the region, 
especially at the school level. 



Services 



In the same period covered by this survey, REL provided about sixty thousand person- 
hours of services: workshops and training, training of trainers, technical assistance, 
conferences and meetings, and collaborative partnerships. Responses to REL services 
were favorable. A large portion of the respondents were familiar with the REL services 
in various forms, with workshops and training as the most recognized form of REL 
services. Over 50 percent of these respondents received the services available to them, 
and most did so more than once. They found REL services to be useful and have noticed 
the positive impact on them, their work and their students’ performance. In addition, 
they found that overall the quality of the services was high. For future REL service 
provision, respondents primarily recommended that REL expand the content coverage of 
its services. 



Workshops 

Workshops and presentations were key components of RJEL services to the region. REL 
conducted 21 of them in nine months, mostly in Honolulu and Saipan, which served as 
the host site of the 16 th Annual Pacific Educational Conference. An overwhelming 
majority of the participants ranked the various attributes of workshops and presentations 
they attended highly, at an average of 4.5 on a five-point scale. The responses indicated 
that the content of the workshops and presentation was appropriate, content coverage 
adequate, and ideas presented useful. It is also clearly stated that the activities met their 
expectations and, as a result, participants would like to recommend REL workshops and 
presentations to their colleagues and friends. A common theme of their comments is to 
allocate more time to these activities since there is so much to learn. 
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In addition to the overwhelmingly positive remarks on REL products and services, survey- 
respondents encouraged and urged REL to keep up with the good work it has been doing 
for the education community throughout the region. Their input for REL service 
provision in the years to come is summarized in the following section. 



Recommendations 

Promote public awareness and improve accessibility of REL products 

An average of 20 percent of the respondents was not familiar with REL products, 
including its most widely distributed newsletters and conference and meeting brochures. 
It is expected that fewer people were familiar with and actually used other types of media 
such as audio/video tapes, CD-ROMs, and websites which require additional hardware 
often not available at many schools in the region. 

Ways should be explored to promote REL products and improve their accessibility to a 
broad base of educators. REL maintains a database with over six thousand mailing 
addresses accumulated over years of consistent efforts. To reach as many target 
audiences as possible, it is essential to update the mailing address regularly, especially 
since the teacher retention rate has been low in the region, including some school districts 
in Hawai‘i. One of these ways would be to seek assistance from new service centers in 
the region that can play a pivotal role in both promoting REL products and getting the 
most current data for updating the REL mailing database. 



Develop workshops and presentations around common themes 

REL services to the region, including products and services, were highly praised 
individually and collectively. However a review of the titles of the products, services, 
and workshops/presentations failed to identify strong common themes linking activities 
together as concerted efforts for many of the services. As suggested by one of the 
participants, REL should, based on the needs in the region and its strengths/specialty 
area, develop series of workshops with a clear focus and coordinate the production and 
dissemination of products and technical assistance. Then REL will have a good chance 
to improve decision-making, school administration, teaching practices, and ultimately 
improve students’ performance. 



Follow-up on the initial services 

Introducing new knowledge and promising practices to educators via workshops and 
presentations or other types of services is the beginning of a long process. To make sure 
that the seeds planted will bear fruits, follow-up activities are an integral part of this 
process. Also, in line with the second recommendation, REL should consider developing 
means to measure the fidelity of implementations and the changes occurring over time 
with respect to specific programs. 
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APPENDIX A: REL Annual Client' Survey of Products and Services 
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October 5, 1 999 



Dear Service Recipient: 

The Regional Educational Laboratory (REL) is a major program of Pacific 
Resources for Education and Learning (PREL). The REL’s goal is to help all 
students meet high standards through development, applied research, 
dissemination, and technical assistance conducted with local, state and 
intermediate agencies. Over the years, the REL has distributed briefing papers, 
research syntheses and reports, audio/video tapes, and other informational 
materials. The REL also has conducted workshops and training, provided 
technical assistance, and sponsored conferences. 

Please take time to fill out this survey of the services and materials provided in 
this contract year from December 1998. The information is important for us. We 
will use it to improve REL’s materials and strengthen its services. 

We request that you complete this survey and send it to us in the enclosed 
Business Reply Mail envelope by Monday, October 18, 1999. Thank you for your 
cooperation and we look forward to serving you again soon. 

Sincerely, 



Ormond Hammond, Ph.D. 
Director, Planning and Evaluation 







PREL 



Regional Educational Laboratory (REL) 

A Program of 

Pacific Resources for Education and Learning (PREL) 
Annual Participant Survey 

September 1999 

This survey is designed to collect information that will help the Regional Educational 
Laboratory (REL) at PREL improve its products and services to you. Please read each item 
carefully and respond as accurately as possible, filling in the appropriate circle completely 
using either a pen or pencil. 






Indicate the State/Entity in which you work: 

O A. Samoa O Chuuk O CNMI O Guam O Hawai'i 

O Kosrae O RMI O Palau O Pohnpei O Yap 

Indicate the one category that most closely describes your professional role: 

O Teacher: 

O Elementary O Middle/Junior High 

O High School O Other (e.g., K- 12) 

O Curriculum/Content Specialist (e.g., central/district specialist, counselor) 

O Administrator (e.g., principal, program director) 

O Other (please specify): 



O Other 



REL Products 

REL provides products that support educational improvement. Please indicate if you are familiar with each of the 
REL products listed below, by filling the circle corresponding to “yes” or “no.” For each product you are familiar 
with, also indicate how often you use it, and the degree to which you find it useful in your work. 



Product j! 


Are you '! 

familiar with 1 j 
this product? 


How frequently have you used this {j 
product in the last year? \\ 


How useful is this product in your work? •} 




Yes 


No »j 


Never 


Once 


2-3 

times 


4 or |* 

more j 1 

times \ l 


Very 

Useful 


Moderately 

Useful 


A 

Little 

Useful 


Not • j 
Useful 


1. Newsletter: Pacific 't 

r Education Updates !. 


0 


{ \ 

o I 


0 


0 


0 


o j' 


0 


0 


0 


0 


2. Briefing papers (e.g., Reading ; 
1 Instruction for Older Struggling j 
Readers) j 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


i 3. Research syntheses and reports 
(e.g., Retention and Attrition of \ 
; Pacific School Teachers and 
i Administrators) ' 


0 


o i 


0 


0 


0 


0 1 


0 


0 


0 


O ;l 


| 4. Audio/video tapes (e.g. First ■ 
■ and Second Language Literacy: j 

From Research to Practice) j 


1 0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


j; 5. Information about conferences i 
and meetings (e.g., Pacific 
! Educational Conference) 


j 0 


0 1 


0 


0 


0 


o ! 


1 0 


0 


0 


0 i! 


i 6. Website 


1 0 


° ! 


0 


0 


0 


0 


j 0 


0 


0 


0 !i 



liRjV: 
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Please respond to the following statements by circling in how strongly you agree or disagree with each. 



Statement 


Strongly 

Agree 


Agree 


Neutral 


Disagree 


Strongly 

Disagree 


Don’t 
Know •] 


■ 1. The quality of RJEL products (e.g., publications and 
? Videotapes) is high. 


0 


0 


0 


0 


0 


0 i 


! 2. REL products have contributed to my professional j 
1 knowledge. | 


0 


0 


0 


0 


0 


o ; 


: 3. REL products have contributed to my professional ] 

| Skills. * 


0 


0 


0 


0 


0 


i 

0 1 


!' 4. REL products have contributed to my professional 5 

$ Work. ] 


0 


0 


0 


0 


0 


0 ! 


( 5. 1 have used REL products to inform decision ; 

| Making and planning. ij 


0 


0 


0 


0 


0 


0 1 


l 6. REL products have positively impacted 
Student performance. 


0 


0 


0 


0 


0 


0 ! 
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REL Services 

REL also provides services that support student educational success. Please indicate whether you are familiar with 
each of the REL services listed below, by circling in “ yes ” or “no.” For each service you are familiar with, also 
indicate how often you received this service, and the degree to which it is useful in your work. 




IerJc 



| Statement j 


Strongly 

Agree 


Agree 


Neutral 


Disagree 


Strongly 

Disagree 


Don’t | 
Know 


\ 1 . The quality of REL services (e.g., technical 
[ assistance, and training) is high. :[ 


0 


0 


0 


0 


0 


0 j 


| 2. REL services have contributed to my professional 

* knowledge. 


0 


0 


0 


0 


0 


0 


;■ 3. REL services have contributed to my 
r professional skills. 


0 


0 


0 


0 


0 


0 p 


i! 4. REL services have contributed to my | 

i; professional work. 


0 


0 


0 


0 


0 


0 


i 5. REL services have impacted my ; 

\ decision making and planning. 


0 


0 


0 


0 


0 


o 


S> 6. REL services have positively impacted student j 

f performance. \ 


; o 


0 


0 


0 


0 


o j 



What do you like best about REL services? 



How could the REL improve its services? 



Educational Reform 

Please respond to the following statements related to the REL’s involvement in educational reforms. 



S Statement ;[ 


Strongly 

Agree 


Agree 


Neutral 


Disagree 


Strongly 

Disagree 


Don’t i 
Know I 


! 1. REL assists schools and school districts in adapting* 
: educational reform strategies that contribute to jj 

: comprehensive reform. |j 


0 


0 


0 


0 


0 


1 

0 ! 


2. The number of schools undergoing comprehensive j! 
educational reforms is increasing over time. !j 


0 


0 


0 


0 


0 


O ! 

| 



Thank you for completing this survey! 

Please use the enclosed, prepaid, return-addressed envelope to return the completed survey by 
October 1, 1999. 
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PACIFIC RESOURCES FOR EDUCATION AND LEARNING 
PresentationA/Vorkshop Participant Questionnaire 



Title/Name of Event: 

Presenter: Date: Entity: 

Which entity are you from? (Check one.) 

| | 1 . American Samoa Q 2. Chuuk 

| | 6. Kosrae Q 7. RMI 

Your professional role. (Check one.) 

| | 1 . Teacher (Elementary) | 1 2. Teacher (Secondary) j |. 3. Principal/Assistant Principal 

| | 4. Curriculum Specialist Q 5. District/Central Administrator Q 6. Resource Teacher 

| | 7. Higher Education Q 8. Parent CD 9. Other (Specify): 

Please evaluate this presentation/workshop by circling the number that best describes your response to 
each of the following questions. 

How appropriate was the content of this workshop for the betterment of Pacific children's education? 



Very appropriate 

5 4 3 2 


Not at all 

1 


How appropriate was the content of this workshop to your work? 




Very appropriate 

5 4 3 2 


Not at all 

1 


How adequate was the content coverage in this workshop? 




Very adequate 

5 4 3 2 


Very inadequate 

1 



□ 3. CNMI 
I | 8. Palau 



| | 4. Guam 
| | 9. Pohnpei 



| | 5. Hawai'i 
I | 10. Yap 



How useful are the ideas and content of this workshop in your work? 

Very useful Not useful 

5 4 3 2 1 

How likely are you to recommend the ideas of this workshop to others? 

Very likely Very unlikely 

5 4 3 2 1 

What impact will the content of this workshop have on the betterment of Pacific children’s education? 

Very useful Not useful 

5 4 3 2 1 



Did this workshop meet your expectations? 








Very useful 






Not useful 


5 4 


3 


2 


1 



How do you feel about the overall quality of this workshop? 

Very useful 

5 4 3 2 

45 




Not useful 

1 



What did you like about the content and/or the style of presentation of this workshop? 



Suggestions regarding the content and/or the style of presentation of this workshop: 



What should be the content of the next workshop as a follow-up to this workshop? 



Additional Comments: 



PREL 2/97 



0 
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Listing of Products Distributed 



REL’s New Products Distributed 



Annual Report: 

e 1998 PREL Annual Report: A Report to the Board of Directors 
0 REL 1998 Annual Report: Improving America’s Schools through Educational R&D 

Brochures and Posters: 

® 16 th Annual PEC Poster 

• 16 th Annual PEC Registration Packet Brochure 

® Classroom Organization and Management Program Flier 

• Discipline-Based Art Education Flier 

® Hawai‘i Art Education Association Flier 
e Literacy Teleconference Video Announcement 
» Pacific Curriculum and Instruction Council Brochure 
« Pacific Regional Educational Laboratory Brochure 

• Pacific Resources for Education and Learning Information Sheet 

• PREL Products and Publications Catalog, 1999-2000 

• Program Planning Council Brochure 

0 Workshop/Conference Information Brochure 

Newsletters: 

• Pacific Center News, Vol. 3, No. 3 
® Pacific Center News, Vol. 4, No. 1 

• Pacific Education Updates, Vol. 10, No. 4 

• Pacific Education Updates, Vol. 11, No. 1 

REL Briefing Paper: 

9 Professional Development for School Leaders -- A Hawai'i Model 
9 Reading Instruction for Older Struggling Readers 

• Title VII Bilingual Education In Hawai’i: Lessons Learned 

Promising Practices in the Pacific Region: 

• Promising Practices in the Pacific Region 1998, Issue 2 

Videotapes: 

9 CRIZMAC: Island Worlds 

• First and Second Language Literacy: From Research to Practice 

• PEC ‘97, Nainoa Thompson Keynote Speech 

Audio tapes: 

• Reading Aloud to Children — Carolinian 

• Reading Aloud to Children — Kosraean 
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Directory: 

® Service Providers’ Networking Directory. 1999 

Focused Discussion Series: 

® Peaceful Schools 

Others: 

» 21 st Century Learning Centers Program Booklets 
® DBAE Material Folders 

• First and Second Language Literacy Teleconference Announcement 



REL's Previous Products Distributed 



Annual Reports: 

• 1996 Annual Report: Pacific Resources for Education and Learning 

• 1997 Annual Report: Pacific Resources for Education and Learning 

• 1997 PREL Annual Report: A Report to the Board of Directors 

• 1998 Annual Report: Pacific Resources for Education and Learning 

Audio Tapes: 

• Reading Aloud to Children (bilingual audio tapes available in eight different Pacific 
languages) 

• Reading Aloud to Children in Chuukese 

• Reading Aloud to Children in Hawaiian 

• Reading Aloud to Children in Marshallese 

• Reading Aloud to Children in Palauan 

• Reading Aloud to Children in Pohnpeian 

• Reading Aloud to Children in Samoan 

• Reading Aloud to Children in Woleaian (Yap) 

• Reading Aloud to Children in Yapese 

Brochures and Posters: 

• 15 th Annual Pacific Educational Conference Registration Packet — Voyaging With a 
Vision 

• 16 th Annual Pacific Educational Conference Registration Packet — Pacific 
Communities: Leaders of The New Millennium 

• 1998 Publications and Products Catalog 

• A Checklist for Helping Your Child with Homework Brochure 

• A Checklist for Helping Your Child with Homework Poster 

• A Checklist for Helping Your Child with Homework Poster — American Samoan 

• A Checklist for Helping Your Child with Homework Poster — Carolinian 

• A Checklist for Helping Your Child with Homework Poster — Chamorro 

• A Checklist for Helping Your Child with Homework Poster — Chuukese 

• A Checklist for Helping Your Child with Homework Poster — English 
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® A Checklist for Helping Your Child with Homework Poster — Marshallese 
® A Checklist for Helping Your Child with Homework Poster — Palauan 
© A Checklist for Helping Your Child with Homework Poster — Pohnpeian 
® A Conversation with Parents about Language — Chinese 
® A Conversation with Parents about Language — Chuukese 
o A Conversation with Parents about Language — Ilocano 
o A Conversation with Parents about Language — Marshallese 
a A Conversation with Parents about Language — Pohnpeian 

• A Conversation with Parents about Language — Vietnamese 
® Adult Literacy brochure 

» Fact Sheet: Comprehensive School Reform Program 

• Let’s Do Homework brochure — American Samoan 

• Let’s Do Homework brochure — Chamorro 

• Let’s Do Homework brochure — English 

• Let’s Do Homework brochure — Marshallese 

• Our Pacific Region brochure 

® Pacific Center for the Arts and Humanities in Education (PCAHE) Brochure 

• Pacific Resources for Education and Learning Information Sheet 

• PEIR Program Brochure 

» PREL Publications and Products, 1998 Catalog 

• Talking Story “Come Join Me On My Back Porch” 

CD-ROMs: 

• Making Schools Work for Every Child 

• Pacific Interactive 1 .0 CD-ROM 

Directories: 

• Directory of Pacific Professionals for Educational Improvement — Second Edition 
® Service Providers’ Networking Directory, 1998 

Educational Innovations in the Pacific: 

• Assessment in the Pacific: The Yap Assessment Model 

• Ka'ahumanu School’s ESL Program: Creative Solutions to Some Nagging Problems 

• Schools-Within-A-School: The Kapa’a Elementary School Model 

• University Of Hawaii’s Early Childhood Visual Arts Cohort Concept 

• Year-round Schools: The Star of the Sea Model 

Newsletters: 

• Pacific Education Updates, Vol. 9, No. 2, July, 1997 

• Pacific Education Updates, Vol. 9, No. 4. November. 1997 

• Pacific Education Updates, Vol. 10, No.l, March, 1998 

• Pacific Education Updates, Vol. 10, No. 2, July, 1998 

• Pacific Education Updates, Vol. 10, No. 3, December, 1998 
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REL Briefing Papers: 

o A Review of “The Use of First and Second Language in Education" 

9 America’s Youth: Managed Care’s Most Valuable Population 
® Charter School Concept Versus School/Community-Based Management 
® Integrated Curriculum — A Reflection of Life 

® Integrating Title IV and Title VII: Using TRIBES to Teach English as a Second 
Language (ESL) 

• Language Change and Language Planning and Policy 

• Language of Instruction: Choices and Consequences 

• Managed Care for Children with Disabilities and the Interface of Legislative Policy 

• New Research on Learning Indicates Need for Cultural Awareness 

• School/Community-Based Management Revisited in the Pacific 

• School-Based Programs to Promote Safety and Civility 

» Schools For Disruptive Students: A Questionable Alternative 
® Storytelling: The Heart and Soul of Education 

• Taking a Canoe to the Moon: Comprehensive Art Education for the Pacific 
® Teacher Education in Hawai‘i 

• The Role of Teacher Mentoring in Educational Reform 

Promising Practices in the Pacific Region: 

® Promising Practices in the Pacific Region, 1996 
9 Promising Practices in the Pacific Region, 1997 
® Promising Practices in the Pacific Region. 1998 

• Promising Practices in the Pacific Region: Parental Involvement in Education 
Research and Development Reports: 

9 A Study of Risk Factors Among High School Students in the Pacific Region 

• Culture and Learning at Home and School: A Study in Kosrae State 
9 Educational Needs Assessment for the Pacific Region: 1993-1994 

9 Pacific Region School Finance and Facilities Study 

• Profile of Pacific Public Schools 

• Retention and Attrition of Pacific School Teachers and Administrators (RA.PSTA) 
Research Syntheses: 

• Family Involvement in Education: A Synthesis of Research for Pacific Educators 

• Language Use at Home and School: A Synthesis of Research for Pacific Educators 

• Modified School Schedules: A Look at the Research in the Pacific 

• Research Review for Inclusive Practices 

• Research Review for the Study of Pacific At-Risk Factors 

• Research Synthesis: At-Risk Teachers 

• Synthesis of the Research on Educational Change 

Videotapes: 

• PEC ‘98 Nainoa Thompson Keynote Speech 
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® Clean Water in the Pacific 
® Discipline Based Arts Education 
® Reading the Wind — Video and Guide 

Others: 

® Hawaii Directory of Informal Education Opportunities 
® Making Schools Work for Every Child 
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